{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "uuid": "f9db1dc0-60de-4114-ac7b-caba1b3a4e63"
   },
   "source": [
    "# 使用Jupyter-Notebook快速搭建文本分类应用\n",
    "\n",
    "这是一篇介绍如何在PAI-DSW里用EasyTransfer平台训练文本分类器的教程。只需要一份配置文件，一份ipynb文件，您就可以完成对原始数据的特征提取，网络构建，损失函数及分类评估/预测的简单调用。运行本DEMO需要如下的配置信息\n",
    "\n",
    "- python 3.6+\n",
    "- tensorflow 1.12+\n",
    "\n",
    "## （一）数据准备\n",
    "下面以一个基于bert的文本分类为例，通过端到端的分布式训练/评估/预测流程，展示平台的易用性。这里的端到端指的是直接读入原始数据就可以训练，而不需要事先转换成Bert特征格式。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "uuid": "c39089e1-d85a-485a-8711-65586b1ddeac"
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-10-27 10:51:04--  https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/tutorial/dsw/train.csv\n",
      "Resolving atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com (atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com)... 47.101.88.27\n",
      "Connecting to atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com (atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com)|47.101.88.27|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 6008185 (5.7M) [text/csv]\n",
      "Saving to: ‘./data/train.csv’\n",
      "\n",
      "100%[======================================>] 6,008,185   11.8MB/s   in 0.5s   \n",
      "\n",
      "2020-10-27 10:51:05 (11.8 MB/s) - ‘./data/train.csv’ saved [6008185/6008185]\n",
      "\n",
      "--2020-10-27 10:51:05--  https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/tutorial/dsw/dev.csv\n",
      "Resolving atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com (atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com)... 47.101.88.27\n",
      "Connecting to atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com (atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com)|47.101.88.27|:443... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 1125139 (1.1M) [text/csv]\n",
      "Saving to: ‘./data/dev.csv’\n",
      "\n",
      "100%[======================================>] 1,125,139   --.-K/s   in 0.04s   \n",
      "\n",
      "2020-10-27 10:51:05 (27.2 MB/s) - ‘./data/dev.csv’ saved [1125139/1125139]\n",
      "\n"
     ]
    }
   ],
   "source": [
    "!mkdir data\n",
    "!wget -O ./data/train.csv https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/tutorial/dsw/train.csv\n",
    "!wget -O ./data/dev.csv https://atp-modelzoo-sh.oss-cn-shanghai.aliyuncs.com/tutorial/dsw/dev.csv"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "uuid": "16ae9b6a-62ad-4816-bc08-14e3f4727e06"
   },
   "outputs": [],
   "source": [
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "uuid": "aa8f56c0-db1c-4d0e-803d-976e33786ef2"
   },
   "outputs": [],
   "source": [
    "train_set = pd.read_csv('./data/train.csv', header=None, delimiter='\\t', encoding='utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "dev_set = pd.read_csv('./data/dev.csv', header=None, delimiter='\\t', encoding='utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "uuid": "7699e1a8-1d2b-4a11-9e17-0700dae18636"
   },
   "outputs": [],
   "source": [
    "train_set.columns = ['label','content']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "uuid": "fe9c6f42-9fd8-4227-9c9e-54a5c6e7ba88"
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>label</th>\n",
       "      <th>content</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>agriculture</td>\n",
       "      <td>加快产城融合 以科技创新引领新城区建设 新城区,城镇化率,中心城区,科技新城,科技创新</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>agriculture</td>\n",
       "      <td>9X10米清雅型别墅，大容量简约民宿风，来自美墅建房的诱惑！ 农村,琉璃瓦,民宿,农村自建房...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "         label                                            content\n",
       "0  agriculture        加快产城融合 以科技创新引领新城区建设 新城区,城镇化率,中心城区,科技新城,科技创新\n",
       "1  agriculture  9X10米清雅型别墅，大容量简约民宿风，来自美墅建房的诱惑！ 农村,琉璃瓦,民宿,农村自建房..."
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set.head(2)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "label      53360\n",
       "content    53360\n",
       "dtype: int64"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "train_set.count()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "0    10000\n",
       "1    10000\n",
       "dtype: int64"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "dev_set.count()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "cellType": "code",
    "uuid": "afb86c76-9082-477c-b93a-c041304cbcf7"
   },
   "source": [
    "## （二）定义配置文件\n",
    "\n",
    "如下是我们easytransfe的配置，比如说predict_checkpoint_path是指定验证集上指标最好的checkpoint的路径。\n",
    "详细配置介绍请看easytransfer文档:https://yuque.antfin-inc.com/pai-innovative-algo/apx4dp/dqy4go"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "uuid": "2f687623-98d6-4394-b8ce-558070dc8a6d"
   },
   "outputs": [],
   "source": [
    "config_json = {\n",
    "    \"worker_hosts\": \"locahost\",\n",
    "    \"task_index\": 1,\n",
    "    \"job_name\": \"chief\",\n",
    "    \"num_gpus\": 1,\n",
    "    \"num_workers\": 1,\n",
    "    \"modelZooBasePath\": \"/home/admin/jupyter/my_model_zoo\",\n",
    "    \"preprocess_config\": {\n",
    "        \"input_schema\": \"label:str:1,content:str:1\",\n",
    "        \"first_sequence\": \"content\",\n",
    "        \"second_sequence\": None,\n",
    "        \"sequence_length\": 16,\n",
    "        \"label_name\": \"label\",\n",
    "        \"label_enumerate_values\": \"tech,finance,entertainment,world,car,culture,sports,military,edu,game,travel,agriculture,house,story,stock\",\n",
    "        \"output_schema\": \"label,predictions\"\n",
    "    },\n",
    "    \"model_config\": {\n",
    "        \"pretrain_model_name_or_path\": \"pai-bert-tiny-zh\",\n",
    "        \"num_labels\": 15\n",
    "    },\n",
    "    \"train_config\": {\n",
    "        \"train_input_fp\": \"./data/train.csv\",\n",
    "        \"train_batch_size\": 2,\n",
    "        \"num_epochs\": 0.01,\n",
    "        \"model_dir\": \"model_dir\",\n",
    "        \"optimizer_config\": {\n",
    "            \"learning_rate\": 1e-5\n",
    "        },\n",
    "        \"distribution_config\": {\n",
    "            \"distribution_strategy\": None\n",
    "        }\n",
    "    },\n",
    "    \"evaluate_config\": {\n",
    "        \"eval_input_fp\": \"./data/dev.csv\",\n",
    "        \"eval_batch_size\": 8\n",
    "    },\n",
    "    \"predict_config\": {\n",
    "        \"predict_checkpoint_path\": \"model_dir/model.ckpt-267\",\n",
    "        \"predict_input_fp\": \"./data/dev.csv\",\n",
    "        \"predict_output_fp\": \"./data/predict.csv\",\n",
    "        \"predict_batch_size\": 1\n",
    "    }\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "uuid": "785e9792-dd80-477b-92c0-0b54e3c94213"
   },
   "source": [
    "##  （三）定义分类应用\n",
    "\n",
    "### 导入easytransfer库文件\n",
    "- base_model: 所有应用都需要继承的父类\n",
    "- Config：用来解析配置文件的父类\n",
    "- layers：基础组件。比如Embedding，Attention等\n",
    "- model_zoo: 管理预训练模型的组件库，通过get_pretrained_model方法可调用bert模型\n",
    "- preprocessors：管理各种应用的预处理逻辑\n",
    "- CSVReader：csv格式的数据读取器\n",
    "- softmax_cross_entropy：用于分类任务的损失函数\n",
    "- classification_eval_metrics：用于分类任务的评估指标，比如Accuracy"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "uuid": "10ccec35-4235-4f4a-96a9-85cced4a1eb5"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/admin/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:516: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:517: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:518: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:519: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:520: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:541: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint8 = np.dtype([(\"qint8\", np.int8, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:542: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint8 = np.dtype([(\"quint8\", np.uint8, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:543: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint16 = np.dtype([(\"qint16\", np.int16, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:544: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_quint16 = np.dtype([(\"quint16\", np.uint16, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:545: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  _np_qint32 = np.dtype([(\"qint32\", np.int32, 1)])\n",
      "/home/admin/.local/lib/python3.6/site-packages/tensorboard/compat/tensorflow_stub/dtypes.py:550: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.\n",
      "  np_resource = np.dtype([(\"resource\", np.ubyte, 1)])\n",
      "WARNING: Logging before flag parsing goes to stderr.\n",
      "W1027 10:51:40.673681 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:22: The name tf.logging.set_verbosity is deprecated. Please use tf.compat.v1.logging.set_verbosity instead.\n",
      "\n",
      "W1027 10:51:40.674461 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:22: The name tf.logging.INFO is deprecated. Please use tf.compat.v1.logging.INFO instead.\n",
      "\n",
      "W1027 10:51:40.674915 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:27: The name tf.set_random_seed is deprecated. Please use tf.compat.v1.set_random_seed instead.\n",
      "\n",
      "W1027 10:51:40.688760 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:28: The name tf.reset_default_graph is deprecated. Please use tf.compat.v1.reset_default_graph instead.\n",
      "\n"
     ]
    }
   ],
   "source": [
    "import tensorflow as tf\n",
    "from easytransfer import base_model, Config\n",
    "from easytransfer import layers\n",
    "from easytransfer import model_zoo\n",
    "from easytransfer import preprocessors\n",
    "from easytransfer.datasets import CSVReader,CSVWriter\n",
    "from easytransfer.losses import softmax_cross_entropy\n",
    "from easytransfer.evaluators import classification_eval_metrics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "uuid": "2db018a2-8824-42e8-b357-91b88e148b0f"
   },
   "source": [
    "## 构图\n",
    "完整的训练/评估/预测/链路，由四个函数构成\n",
    "- build_logits: 构图\n",
    "- build_loss：定义损失函数\n",
    "- build_eval_metrics：定义评估指标\n",
    "- build_predictions：定义预测输出"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {
    "uuid": "91c6db6b-425d-4047-9ebe-f0917c1a7aad"
   },
   "outputs": [],
   "source": [
    "class TextClassification(base_model):\n",
    "\n",
    "    def __init__(self, **kwargs):\n",
    "        super(TextClassification, self).__init__(**kwargs)\n",
    "        self.user_defined_config = kwargs[\"user_defined_config\"]\n",
    "\n",
    "    def build_logits(self, features, mode=None):\n",
    "        # 负责对原始数据进行预处理，生成模型需要的特征，比如：input_ids, input_mask, segment_ids等\n",
    "        preprocessor = preprocessors.get_preprocessor(self.pretrain_model_name_or_path,\n",
    "                                                      user_defined_config=self.user_defined_config)\n",
    "\n",
    "        # 负责构建网络的backbone\n",
    "        model = model_zoo.get_pretrained_model(self.pretrain_model_name_or_path)\n",
    "\n",
    "        dense = layers.Dense(self.num_labels, kernel_initializer=layers.get_initializer(0.02), name='dense')\n",
    "\n",
    "        input_ids, input_mask, segment_ids, label_ids = preprocessor(features)\n",
    "\n",
    "        _, pooled_output = model([input_ids, input_mask, segment_ids], mode=mode)\n",
    "\n",
    "        logits = dense(pooled_output)\n",
    "\n",
    "        return logits, label_ids\n",
    "\n",
    "    def build_loss(self, logits, labels):\n",
    "        return softmax_cross_entropy(labels, self.num_labels, logits)\n",
    "    \n",
    "    def build_eval_metrics(self, logits, labels):\n",
    "        return classification_eval_metrics(logits, labels, self.num_labels)\n",
    "\n",
    "    def build_predictions(self, output):\n",
    "        logits, _ = output\n",
    "        predictions = dict()\n",
    "        predictions[\"predictions\"] = tf.argmax(logits, axis=-1, output_type=tf.int32)\n",
    "        return predictions"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "uuid": "c31301bb-0dc0-4506-9c94-73d76942c782"
   },
   "source": [
    "# (四）启动训练"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {
    "uuid": "32cfce03-786a-434f-9166-7b3649e4f84c"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1027 10:51:47.386891 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:62: The name tf.logging.info is deprecated. Please use tf.compat.v1.logging.info instead.\n",
      "\n",
      "I1027 10:51:47.387405 139683208804160 model.py:62] ***************** modelZooBasePath /home/admin/jupyter/my_model_zoo ***************\n"
     ]
    }
   ],
   "source": [
    "config = Config(mode=\"train_and_evaluate_on_the_fly\", config_json=config_json)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {
    "uuid": "ec5ad940-9410-4d95-8a17-788c826f18eb"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "W1027 10:51:47.413406 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:731: The name tf.gfile.Open is deprecated. Please use tf.io.gfile.GFile instead.\n",
      "\n",
      "I1027 10:51:47.597829 139683208804160 model.py:736] total number of training examples 53360\n",
      "I1027 10:51:47.598403 139683208804160 model.py:239] ***********Running in train_and_evaluate_on_the_fly mode***********\n",
      "I1027 10:51:47.598751 139683208804160 model.py:248] ***********Disable Tao***********\n",
      "I1027 10:51:47.599076 139683208804160 model.py:255] ***********Disable AUTO_MIXED_PRECISION***********\n",
      "I1027 10:51:47.599404 139683208804160 model.py:266] ***********NCCL_MAX_NRINGS 4***********\n",
      "I1027 10:51:47.599712 139683208804160 model.py:267] ***********NCCL_MIN_NRINGS 4***********\n",
      "I1027 10:51:47.600023 139683208804160 model.py:268] ***********TF_JIT_PROFILING False***********\n",
      "I1027 10:51:47.600324 139683208804160 model.py:269] ***********PAI_ENABLE_HLO_DUMPER False***********\n",
      "I1027 10:51:47.600628 139683208804160 model.py:356] ***********Single worker, Single gpu, Don't use distribution strategy***********\n",
      "I1027 10:51:47.600950 139683208804160 model.py:386] model_dir: model_dir\n",
      "I1027 10:51:47.601254 139683208804160 model.py:387] num workers: 1\n",
      "I1027 10:51:47.601554 139683208804160 model.py:388] num gpus: 1\n",
      "I1027 10:51:47.601888 139683208804160 model.py:389] learning rate: 1e-05\n",
      "I1027 10:51:47.602177 139683208804160 model.py:390] train batch size: 2\n",
      "I1027 10:51:47.602481 139683208804160 model.py:391] global batch size: 2\n",
      "I1027 10:51:47.602783 139683208804160 model.py:392] num accumulated batches: 1\n",
      "I1027 10:51:47.603082 139683208804160 model.py:393] num model replica: 1\n",
      "I1027 10:51:47.603379 139683208804160 model.py:394] num train examples per epoch: 53360\n",
      "I1027 10:51:47.603678 139683208804160 model.py:395] num epochs: 0.01\n",
      "I1027 10:51:47.603984 139683208804160 model.py:396] train steps: 267\n",
      "I1027 10:51:47.604272 139683208804160 model.py:397] save steps: 26680\n",
      "I1027 10:51:47.604565 139683208804160 model.py:398] throttle secs: 100\n",
      "I1027 10:51:47.604885 139683208804160 model.py:399] keep checkpoint max: 10\n",
      "I1027 10:51:47.605179 139683208804160 model.py:400] warmup ratio: 0.1\n",
      "I1027 10:51:47.605481 139683208804160 model.py:401] gradient clip: True\n",
      "I1027 10:51:47.605794 139683208804160 model.py:402] log step count steps: 100\n",
      "W1027 10:51:47.606146 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:460: The name tf.ConfigProto is deprecated. Please use tf.compat.v1.ConfigProto instead.\n",
      "\n",
      "W1027 10:51:47.606470 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:465: The name tf.GPUOptions is deprecated. Please use tf.compat.v1.GPUOptions instead.\n",
      "\n",
      "I1027 10:51:47.607270 139683208804160 estimator.py:209] Using config: {'_model_dir': 'model_dir', '_tf_random_seed': 123123, '_save_summary_steps': 100, '_save_checkpoints_steps': 26680, '_save_checkpoints_secs': None, '_session_config': intra_op_parallelism_threads: 1024\n",
      "inter_op_parallelism_threads: 1024\n",
      "gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "  allow_growth: true\n",
      "  force_gpu_compatible: true\n",
      "}\n",
      "allow_soft_placement: true\n",
      ", '_keep_checkpoint_max': 10, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f09c88a3470>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "W1027 10:51:47.608236 139683208804160 model_fn.py:630] Estimator's model_fn (<function EzTransEstimator._build_model_fn.<locals>.model_fn at 0x7f09c88a5400>) includes params argument, but params are not passed to Estimator.\n",
      "I1027 10:51:47.608956 139683208804160 model.py:426] num eval steps: None\n"
     ]
    }
   ],
   "source": [
    "app = TextClassification(user_defined_config=config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {
    "uuid": "d2578057-a24c-4056-a928-832b0e2f3024"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1027 10:51:47.652350 139683208804160 reader.py:78] num_parallel_batches 1\n",
      "I1027 10:51:47.652874 139683208804160 reader.py:79] shuffle_buffer_size None\n",
      "I1027 10:51:47.653255 139683208804160 reader.py:80] prefetch_buffer_size 1\n",
      "I1027 10:51:47.653590 139683208804160 reader.py:81] batch_size 2\n",
      "I1027 10:51:47.653918 139683208804160 reader.py:82] distribution_strategy None\n",
      "I1027 10:51:47.654216 139683208804160 reader.py:83] num_micro_batches 1\n",
      "I1027 10:51:47.655252 139683208804160 reader.py:84] input_schema label:str:1,content:str:1\n",
      "I1027 10:51:47.842130 139683208804160 csv_reader.py:54] ./data/train.csv, total number of training examples 53360\n",
      "I1027 10:51:47.842768 139683208804160 reader.py:78] num_parallel_batches 1\n",
      "I1027 10:51:47.843145 139683208804160 reader.py:79] shuffle_buffer_size None\n",
      "I1027 10:51:47.843453 139683208804160 reader.py:80] prefetch_buffer_size 1\n",
      "I1027 10:51:47.843765 139683208804160 reader.py:81] batch_size 8\n",
      "I1027 10:51:47.844077 139683208804160 reader.py:82] distribution_strategy None\n",
      "I1027 10:51:47.844377 139683208804160 reader.py:83] num_micro_batches 1\n",
      "I1027 10:51:47.844685 139683208804160 reader.py:84] input_schema label:str:1,content:str:1\n",
      "I1027 10:51:47.884038 139683208804160 csv_reader.py:59] ./data/dev.csv, total number of eval examples 10000\n"
     ]
    }
   ],
   "source": [
    "train_reader = CSVReader(input_glob=app.train_input_fp,\n",
    "                         is_training=True,\n",
    "                         input_schema=app.input_schema,\n",
    "                         batch_size=app.train_batch_size)\n",
    "\n",
    "eval_reader = CSVReader(input_glob=app.eval_input_fp,\n",
    "                        is_training=False,\n",
    "                        input_schema=app.input_schema,\n",
    "                        batch_size=app.eval_batch_size)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {
    "uuid": "84bb64e6-5461-4685-b105-872c6ad01c27"
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1027 10:51:47.915409 139683208804160 estimator_training.py:186] Not using Distribute Coordinator.\n",
      "I1027 10:51:47.915982 139683208804160 training.py:612] Running training and evaluation locally (non-distributed).\n",
      "I1027 10:51:47.916495 139683208804160 training.py:700] Start train and evaluate loop. The evaluate will happen after every checkpoint. Checkpoint frequency is determined based on RunConfig arguments: save_checkpoints_steps 26680 or save_checkpoints_secs None.\n",
      "W1027 10:51:47.923486 139683208804160 deprecation.py:323] From /home/admin/.local/lib/python3.6/site-packages/tensorflow/python/training/training_util.py:236: Variable.initialized_value (from tensorflow.python.ops.variables) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use Variable.read_value. Variables in 2.X are initialized automatically both in eager and graph (inside tf.defun) contexts.\n",
      "I1027 10:51:47.946563 139683208804160 reader.py:89] Random shuffle on the whole 53360 training examples\n",
      "W1027 10:51:47.952159 139683208804160 deprecation.py:323] From /home/admin/.local/lib/python3.6/site-packages/tensorflow/python/data/util/random_seed.py:58: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use tf.where in 2.0, which has the same broadcast rule as np.where\n",
      "W1027 10:51:47.954110 139683208804160 deprecation.py:323] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/datasets/reader.py:104: map_and_batch (from tensorflow.python.data.experimental.ops.batching) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use `tf.data.Dataset.map(map_func, num_parallel_calls)` followed by `tf.data.Dataset.batch(batch_size, drop_remainder)`. Static tf.data optimizations will take care of using the fused implementation.\n",
      "W1027 10:51:47.955716 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/datasets/csv_reader.py:90: The name tf.decode_csv is deprecated. Please use tf.io.decode_csv instead.\n",
      "\n",
      "I1027 10:51:47.969672 139683208804160 estimator.py:1145] Calling model_fn.\n",
      "I1027 10:51:47.991189 139683208804160 preprocessor.py:127] ********** Begin to download to /home/admin/jupyter/my_model_zoo/bert/pai-bert-tiny-zh.tgz **********\n",
      "W1027 10:51:52.208888 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/preprocessors/preprocessor.py:143: The name tf.gfile.Exists is deprecated. Please use tf.io.gfile.exists instead.\n",
      "\n",
      "W1027 10:51:52.210010 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/preprocessors/preprocessor.py:144: The name tf.gfile.MakeDirs is deprecated. Please use tf.io.gfile.makedirs instead.\n",
      "\n",
      "W1027 10:51:52.220342 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/preprocessors/preprocessor.py:147: The name tf.gfile.Copy is deprecated. Please use tf.io.gfile.copy instead.\n",
      "\n",
      "W1027 10:51:52.272022 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/preprocessors/tokenization.py:116: The name tf.gfile.GFile is deprecated. Please use tf.io.gfile.GFile instead.\n",
      "\n",
      "W1027 10:51:52.352205 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/layers/utils.py:102: The name tf.keras.initializers.TruncatedNormal is deprecated. Please use tf.compat.v1.keras.initializers.TruncatedNormal instead.\n",
      "\n",
      "W1027 10:51:52.352815 139683208804160 deprecation.py:506] From /home/admin/.local/lib/python3.6/site-packages/tensorflow/python/keras/initializers.py:94: calling TruncatedNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Call initializer instance with the dtype argument instead of passing it to the constructor\n",
      "W1027 10:52:12.478384 139683208804160 lazy_loader.py:50] \n",
      "The TensorFlow contrib module will not be included in TensorFlow 2.0.\n",
      "For more information, please see:\n",
      "  * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md\n",
      "  * https://github.com/tensorflow/addons\n",
      "  * https://github.com/tensorflow/io (for I/O related ops)\n",
      "If you depend on functionality not listed there, please file an issue.\n",
      "\n",
      "I1027 10:52:14.657708 139683208804160 modeling_utils.py:156] Load weights from /home/admin/jupyter/my_model_zoo/bert/pai-bert-tiny-zh/model.ckpt\n",
      "W1027 10:52:14.658426 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/model_zoo/modeling_utils.py:157: The name tf.train.init_from_checkpoint is deprecated. Please use tf.compat.v1.train.init_from_checkpoint instead.\n",
      "\n",
      "W1027 10:52:15.139382 139683208804160 deprecation.py:323] From /home/admin/.local/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py:255: py_func (from tensorflow.python.ops.script_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "tf.py_func is deprecated in TF V2. Instead, there are two\n",
      "    options available in V2.\n",
      "    - tf.py_function takes a python function which manipulates tf eager\n",
      "    tensors instead of numpy arrays. It's easy to convert a tf eager tensor to\n",
      "    an ndarray (just call tensor.numpy()) but having access to eager tensors\n",
      "    means `tf.py_function`s can use accelerators such as GPUs as well as\n",
      "    being differentiable using a gradient tape.\n",
      "    - tf.numpy_function maintains the semantics of the deprecated tf.py_func\n",
      "    (it is not differentiable, and manipulates numpy arrays). It drops the\n",
      "    stateful argument making all functions stateful.\n",
      "    \n",
      "W1027 10:52:15.142670 139683208804160 deprecation.py:506] From /home/admin/.local/lib/python3.6/site-packages/tensorflow/python/autograph/impl/api.py:253: calling string_split (from tensorflow.python.ops.ragged.ragged_string_ops) with delimiter is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "delimiter is deprecated, please use sep instead.\n",
      "W1027 10:52:15.756582 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/losses/classification_regression_loss.py:22: The name tf.losses.softmax_cross_entropy is deprecated. Please use tf.compat.v1.losses.softmax_cross_entropy instead.\n",
      "\n",
      "W1027 10:52:15.790305 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/optimizers/__init__.py:37: The name tf.train.get_or_create_global_step is deprecated. Please use tf.compat.v1.train.get_or_create_global_step instead.\n",
      "\n",
      "W1027 10:52:15.790932 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/optimizers/__init__.py:39: The name tf.train.polynomial_decay is deprecated. Please use tf.compat.v1.train.polynomial_decay instead.\n",
      "\n",
      "W1027 10:52:15.795329 139683208804160 deprecation.py:323] From /home/admin/.local/lib/python3.6/site-packages/tensorflow/python/keras/optimizer_v2/learning_rate_schedule.py:409: div (from tensorflow.python.ops.math_ops) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Deprecated in favor of operator or tf.math.divide.\n",
      "I1027 10:52:15.799872 139683208804160 __init__.py:50] *******Warmup 26 steps***********\n",
      "I1027 10:52:15.806827 139683208804160 __init__.py:77] *******Using adam optimizer************\n",
      "W1027 10:52:15.807247 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/optimizers/__init__.py:78: The name tf.train.AdamOptimizer is deprecated. Please use tf.compat.v1.train.AdamOptimizer instead.\n",
      "\n",
      "I1027 10:52:16.699470 139683208804160 __init__.py:100] *******Num of trainable variables 3223577************\n",
      "I1027 10:52:16.700337 139683208804160 __init__.py:103] *******Clip Gradients************\n",
      "I1027 10:52:16.700689 139683208804160 __init__.py:104] *******Clip Norm Value 1.0*********\n",
      "W1027 10:52:17.177729 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/optimizers/__init__.py:122: The name tf.summary.scalar is deprecated. Please use tf.compat.v1.summary.scalar instead.\n",
      "\n",
      "W1027 10:52:17.179639 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/utils/hooks.py:20: The name tf.train.SessionRunHook is deprecated. Please use tf.estimator.SessionRunHook instead.\n",
      "\n",
      "W1027 10:52:17.180127 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/utils/hooks.py:27: The name tf.summary.FileWriter is deprecated. Please use tf.compat.v1.summary.FileWriter instead.\n",
      "\n",
      "W1027 10:52:17.186133 139683208804160 deprecation_wrapper.py:119] From /home/admin/.local/lib/python3.6/site-packages/easytransfer/engines/model.py:519: The name tf.summary.merge_all is deprecated. Please use tf.compat.v1.summary.merge_all instead.\n",
      "\n",
      "I1027 10:52:17.187418 139683208804160 estimator.py:1147] Done calling model_fn.\n",
      "I1027 10:52:17.189625 139683208804160 basic_session_run_hooks.py:541] Create CheckpointSaverHook.\n",
      "I1027 10:52:18.032090 139683208804160 monitored_session.py:240] Graph was finalized.\n",
      "I1027 10:52:18.937041 139683208804160 session_manager.py:500] Running local_init_op.\n",
      "I1027 10:52:18.980668 139683208804160 session_manager.py:502] Done running local_init_op.\n",
      "I1027 10:52:20.671645 139683208804160 basic_session_run_hooks.py:606] Saving checkpoints for 0 into model_dir/model.ckpt.\n",
      "I1027 10:52:22.754931 139683208804160 basic_session_run_hooks.py:262] loss = 2.639944, step = 1\n",
      "I1027 10:52:22.756044 139683208804160 hooks.py:51] progress = 0.00%, avg_loss = 2.639944\n",
      "I1027 10:52:31.882585 139683208804160 basic_session_run_hooks.py:692] global_step/sec: 10.9554\n",
      "I1027 10:52:31.883865 139683208804160 basic_session_run_hooks.py:260] loss = 2.734365, step = 101 (9.129 sec)\n",
      "I1027 10:52:31.884430 139683208804160 hooks.py:51] progress = 37.45%, avg_loss = 2.681827\n",
      "I1027 10:52:39.470747 139683208804160 basic_session_run_hooks.py:692] global_step/sec: 13.1784\n",
      "I1027 10:52:39.472079 139683208804160 basic_session_run_hooks.py:260] loss = 2.794959, step = 201 (7.588 sec)\n",
      "I1027 10:52:39.472646 139683208804160 hooks.py:51] progress = 74.91%, avg_loss = 2.693605\n",
      "I1027 10:52:44.782931 139683208804160 basic_session_run_hooks.py:606] Saving checkpoints for 267 into model_dir/model.ckpt.\n",
      "I1027 10:52:45.310904 139683208804160 estimator.py:1145] Calling model_fn.\n",
      "I1027 10:52:45.903835 139683208804160 modeling_utils.py:156] Load weights from /home/admin/jupyter/my_model_zoo/bert/pai-bert-tiny-zh/model.ckpt\n",
      "I1027 10:52:46.664612 139683208804160 classification_regression_evaluator.py:47] empty data to evaluate\n",
      "I1027 10:52:46.671829 139683208804160 estimator.py:1147] Done calling model_fn.\n",
      "I1027 10:52:46.689046 139683208804160 evaluation.py:255] Starting evaluation at 2020-10-27T10:52:46Z\n",
      "I1027 10:52:46.839423 139683208804160 monitored_session.py:240] Graph was finalized.\n",
      "W1027 10:52:46.840441 139683208804160 deprecation.py:323] From /home/admin/.local/lib/python3.6/site-packages/tensorflow/python/training/saver.py:1276: checkpoint_exists (from tensorflow.python.training.checkpoint_management) is deprecated and will be removed in a future version.\n",
      "Instructions for updating:\n",
      "Use standard file APIs to check for files with this prefix.\n",
      "I1027 10:52:46.843447 139683208804160 saver.py:1280] Restoring parameters from model_dir/model.ckpt-267\n",
      "I1027 10:52:47.097299 139683208804160 session_manager.py:500] Running local_init_op.\n",
      "I1027 10:52:47.117353 139683208804160 session_manager.py:502] Done running local_init_op.\n",
      "I1027 10:53:11.988628 139683208804160 evaluation.py:275] Finished evaluation at 2020-10-27-10:53:11\n",
      "I1027 10:53:11.989410 139683208804160 estimator.py:2039] Saving dict for global step 267: global_step = 267, loss = 2.7040272, py_accuracy = 0.0763, py_macro_f1 = 0.029714607, py_micro_f1 = 0.0763, py_weighted_f1 = 0.03433238\n",
      "I1027 10:53:12.204726 139683208804160 estimator.py:2099] Saving 'checkpoint_path' summary for global step 267: model_dir/model.ckpt-267\n",
      "I1027 10:53:12.258248 139683208804160 estimator.py:368] Loss for final step: 2.625153.\n"
     ]
    }
   ],
   "source": [
    "app.run_train_and_evaluate(train_reader=train_reader, eval_reader=eval_reader)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# (五）启动预测"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "I1027 10:53:17.824488 139683208804160 model.py:62] ***************** modelZooBasePath /home/admin/jupyter/my_model_zoo ***************\n",
      "I1027 10:53:17.865181 139683208804160 model.py:772] total number of predicting examples 10000\n",
      "I1027 10:53:17.865805 139683208804160 model.py:437] ***********Running in predict_on_the_fly mode***********\n",
      "W1027 10:53:17.866708 139683208804160 estimator.py:1811] Using temporary folder as model directory: /tmp/tmpus3ibqy0\n",
      "I1027 10:53:17.867390 139683208804160 estimator.py:209] Using config: {'_model_dir': '/tmp/tmpus3ibqy0', '_tf_random_seed': None, '_save_summary_steps': 100, '_save_checkpoints_steps': None, '_save_checkpoints_secs': 600, '_session_config': intra_op_parallelism_threads: 1024\n",
      "inter_op_parallelism_threads: 1024\n",
      "gpu_options {\n",
      "  per_process_gpu_memory_fraction: 1.0\n",
      "  allow_growth: true\n",
      "  force_gpu_compatible: true\n",
      "}\n",
      "allow_soft_placement: true\n",
      ", '_keep_checkpoint_max': 5, '_keep_checkpoint_every_n_hours': 10000, '_log_step_count_steps': 100, '_train_distribute': None, '_device_fn': None, '_protocol': None, '_eval_distribute': None, '_experimental_distribute': None, '_experimental_max_worker_delay_secs': None, '_service': None, '_cluster_spec': <tensorflow.python.training.server_lib.ClusterSpec object at 0x7f09b1e59160>, '_task_type': 'worker', '_task_id': 0, '_global_id_in_cluster': 0, '_master': '', '_evaluation_master': '', '_is_chief': True, '_num_ps_replicas': 0, '_num_worker_replicas': 1}\n",
      "W1027 10:53:17.867870 139683208804160 model_fn.py:630] Estimator's model_fn (<function EzTransEstimator._build_model_fn.<locals>.model_fn at 0x7f09680b4ae8>) includes params argument, but params are not passed to Estimator.\n",
      "I1027 10:53:17.868389 139683208804160 reader.py:78] num_parallel_batches 1\n",
      "I1027 10:53:17.868743 139683208804160 reader.py:79] shuffle_buffer_size None\n",
      "I1027 10:53:17.869065 139683208804160 reader.py:80] prefetch_buffer_size 1\n",
      "I1027 10:53:17.869393 139683208804160 reader.py:81] batch_size 1\n",
      "I1027 10:53:17.869695 139683208804160 reader.py:82] distribution_strategy None\n",
      "I1027 10:53:17.870013 139683208804160 reader.py:83] num_micro_batches 1\n",
      "I1027 10:53:17.870311 139683208804160 reader.py:84] input_schema label:str:1,content:str:1\n",
      "I1027 10:53:17.911736 139683208804160 csv_reader.py:59] ./data/dev.csv, total number of eval examples 10000\n",
      "I1027 10:53:17.942214 139683208804160 estimator.py:1145] Calling model_fn.\n",
      "I1027 10:53:18.818063 139683208804160 modeling_utils.py:156] Load weights from /home/admin/jupyter/my_model_zoo/bert/pai-bert-tiny-zh/model.ckpt\n",
      "I1027 10:53:19.499733 139683208804160 estimator.py:1147] Done calling model_fn.\n",
      "I1027 10:53:19.640666 139683208804160 monitored_session.py:240] Graph was finalized.\n",
      "I1027 10:53:19.648801 139683208804160 saver.py:1280] Restoring parameters from model_dir/model.ckpt-267\n",
      "I1027 10:53:19.856720 139683208804160 session_manager.py:500] Running local_init_op.\n",
      "I1027 10:53:19.872511 139683208804160 session_manager.py:502] Done running local_init_op.\n",
      "I1027 10:53:20.159153 139683208804160 model.py:595] Processing 0 batches\n",
      "I1027 10:53:20.519963 139683208804160 model.py:595] Processing 100 batches\n",
      "I1027 10:53:20.865264 139683208804160 model.py:595] Processing 200 batches\n",
      "I1027 10:53:21.214265 139683208804160 model.py:595] Processing 300 batches\n",
      "I1027 10:53:21.553737 139683208804160 model.py:595] Processing 400 batches\n",
      "I1027 10:53:21.893859 139683208804160 model.py:595] Processing 500 batches\n",
      "I1027 10:53:22.233795 139683208804160 model.py:595] Processing 600 batches\n",
      "I1027 10:53:22.574395 139683208804160 model.py:595] Processing 700 batches\n",
      "I1027 10:53:22.910066 139683208804160 model.py:595] Processing 800 batches\n",
      "I1027 10:53:23.246499 139683208804160 model.py:595] Processing 900 batches\n",
      "I1027 10:53:23.590281 139683208804160 model.py:595] Processing 1000 batches\n",
      "I1027 10:53:23.942818 139683208804160 model.py:595] Processing 1100 batches\n",
      "I1027 10:53:24.292021 139683208804160 model.py:595] Processing 1200 batches\n",
      "I1027 10:53:24.635029 139683208804160 model.py:595] Processing 1300 batches\n",
      "I1027 10:53:24.978806 139683208804160 model.py:595] Processing 1400 batches\n",
      "I1027 10:53:25.319749 139683208804160 model.py:595] Processing 1500 batches\n",
      "I1027 10:53:25.666871 139683208804160 model.py:595] Processing 1600 batches\n",
      "I1027 10:53:26.011619 139683208804160 model.py:595] Processing 1700 batches\n",
      "I1027 10:53:26.351907 139683208804160 model.py:595] Processing 1800 batches\n",
      "I1027 10:53:26.694909 139683208804160 model.py:595] Processing 1900 batches\n",
      "I1027 10:53:27.034503 139683208804160 model.py:595] Processing 2000 batches\n",
      "I1027 10:53:27.382215 139683208804160 model.py:595] Processing 2100 batches\n",
      "I1027 10:53:27.729818 139683208804160 model.py:595] Processing 2200 batches\n",
      "I1027 10:53:28.068351 139683208804160 model.py:595] Processing 2300 batches\n",
      "I1027 10:53:28.404124 139683208804160 model.py:595] Processing 2400 batches\n",
      "I1027 10:53:28.742104 139683208804160 model.py:595] Processing 2500 batches\n",
      "I1027 10:53:29.080809 139683208804160 model.py:595] Processing 2600 batches\n",
      "I1027 10:53:29.415561 139683208804160 model.py:595] Processing 2700 batches\n",
      "I1027 10:53:29.753279 139683208804160 model.py:595] Processing 2800 batches\n",
      "I1027 10:53:30.088920 139683208804160 model.py:595] Processing 2900 batches\n",
      "I1027 10:53:30.427825 139683208804160 model.py:595] Processing 3000 batches\n",
      "I1027 10:53:30.764617 139683208804160 model.py:595] Processing 3100 batches\n",
      "I1027 10:53:31.099409 139683208804160 model.py:595] Processing 3200 batches\n",
      "I1027 10:53:31.434019 139683208804160 model.py:595] Processing 3300 batches\n",
      "I1027 10:53:31.769217 139683208804160 model.py:595] Processing 3400 batches\n",
      "I1027 10:53:32.115402 139683208804160 model.py:595] Processing 3500 batches\n",
      "I1027 10:53:32.449070 139683208804160 model.py:595] Processing 3600 batches\n",
      "I1027 10:53:32.785187 139683208804160 model.py:595] Processing 3700 batches\n",
      "I1027 10:53:33.126013 139683208804160 model.py:595] Processing 3800 batches\n",
      "I1027 10:53:33.461882 139683208804160 model.py:595] Processing 3900 batches\n",
      "I1027 10:53:33.799950 139683208804160 model.py:595] Processing 4000 batches\n",
      "I1027 10:53:34.144973 139683208804160 model.py:595] Processing 4100 batches\n",
      "I1027 10:53:34.481841 139683208804160 model.py:595] Processing 4200 batches\n",
      "I1027 10:53:34.821582 139683208804160 model.py:595] Processing 4300 batches\n",
      "I1027 10:53:35.159927 139683208804160 model.py:595] Processing 4400 batches\n",
      "I1027 10:53:35.504789 139683208804160 model.py:595] Processing 4500 batches\n",
      "I1027 10:53:35.847730 139683208804160 model.py:595] Processing 4600 batches\n",
      "I1027 10:53:36.199033 139683208804160 model.py:595] Processing 4700 batches\n",
      "I1027 10:53:36.539824 139683208804160 model.py:595] Processing 4800 batches\n",
      "I1027 10:53:36.876594 139683208804160 model.py:595] Processing 4900 batches\n",
      "I1027 10:53:37.212835 139683208804160 model.py:595] Processing 5000 batches\n",
      "I1027 10:53:37.547269 139683208804160 model.py:595] Processing 5100 batches\n",
      "I1027 10:53:37.882551 139683208804160 model.py:595] Processing 5200 batches\n",
      "I1027 10:53:38.218382 139683208804160 model.py:595] Processing 5300 batches\n",
      "I1027 10:53:38.557312 139683208804160 model.py:595] Processing 5400 batches\n",
      "I1027 10:53:38.897628 139683208804160 model.py:595] Processing 5500 batches\n",
      "I1027 10:53:39.239755 139683208804160 model.py:595] Processing 5600 batches\n",
      "I1027 10:53:39.584638 139683208804160 model.py:595] Processing 5700 batches\n",
      "I1027 10:53:39.922549 139683208804160 model.py:595] Processing 5800 batches\n",
      "I1027 10:53:40.263690 139683208804160 model.py:595] Processing 5900 batches\n",
      "I1027 10:53:40.603480 139683208804160 model.py:595] Processing 6000 batches\n",
      "I1027 10:53:40.941608 139683208804160 model.py:595] Processing 6100 batches\n",
      "I1027 10:53:41.280839 139683208804160 model.py:595] Processing 6200 batches\n",
      "I1027 10:53:41.617161 139683208804160 model.py:595] Processing 6300 batches\n",
      "I1027 10:53:41.958600 139683208804160 model.py:595] Processing 6400 batches\n",
      "I1027 10:53:42.297745 139683208804160 model.py:595] Processing 6500 batches\n",
      "I1027 10:53:42.645255 139683208804160 model.py:595] Processing 6600 batches\n",
      "I1027 10:53:42.984472 139683208804160 model.py:595] Processing 6700 batches\n",
      "I1027 10:53:43.319318 139683208804160 model.py:595] Processing 6800 batches\n",
      "I1027 10:53:43.653437 139683208804160 model.py:595] Processing 6900 batches\n",
      "I1027 10:53:43.994092 139683208804160 model.py:595] Processing 7000 batches\n",
      "I1027 10:53:44.342529 139683208804160 model.py:595] Processing 7100 batches\n",
      "I1027 10:53:44.688382 139683208804160 model.py:595] Processing 7200 batches\n",
      "I1027 10:53:45.022351 139683208804160 model.py:595] Processing 7300 batches\n",
      "I1027 10:53:45.364918 139683208804160 model.py:595] Processing 7400 batches\n",
      "I1027 10:53:45.702715 139683208804160 model.py:595] Processing 7500 batches\n",
      "I1027 10:53:46.040739 139683208804160 model.py:595] Processing 7600 batches\n",
      "I1027 10:53:46.377025 139683208804160 model.py:595] Processing 7700 batches\n",
      "I1027 10:53:47.051711 139683208804160 model.py:595] Processing 7900 batches\n",
      "I1027 10:53:47.401490 139683208804160 model.py:595] Processing 8000 batches\n",
      "I1027 10:53:47.743407 139683208804160 model.py:595] Processing 8100 batches\n",
      "I1027 10:53:48.080727 139683208804160 model.py:595] Processing 8200 batches\n",
      "I1027 10:53:48.416368 139683208804160 model.py:595] Processing 8300 batches\n",
      "I1027 10:53:48.754052 139683208804160 model.py:595] Processing 8400 batches\n",
      "I1027 10:53:49.095826 139683208804160 model.py:595] Processing 8500 batches\n",
      "I1027 10:53:49.447168 139683208804160 model.py:595] Processing 8600 batches\n",
      "I1027 10:53:49.787862 139683208804160 model.py:595] Processing 8700 batches\n",
      "I1027 10:53:50.136596 139683208804160 model.py:595] Processing 8800 batches\n",
      "I1027 10:53:50.479019 139683208804160 model.py:595] Processing 8900 batches\n",
      "I1027 10:53:50.822014 139683208804160 model.py:595] Processing 9000 batches\n",
      "I1027 10:53:51.167615 139683208804160 model.py:595] Processing 9100 batches\n",
      "I1027 10:53:51.514851 139683208804160 model.py:595] Processing 9200 batches\n",
      "I1027 10:53:51.856305 139683208804160 model.py:595] Processing 9300 batches\n",
      "I1027 10:53:52.195043 139683208804160 model.py:595] Processing 9400 batches\n",
      "I1027 10:53:52.534734 139683208804160 model.py:595] Processing 9500 batches\n",
      "I1027 10:53:52.878989 139683208804160 model.py:595] Processing 9600 batches\n",
      "I1027 10:53:53.223351 139683208804160 model.py:595] Processing 9700 batches\n",
      "I1027 10:53:53.566443 139683208804160 model.py:595] Processing 9800 batches\n",
      "I1027 10:53:53.915706 139683208804160 model.py:595] Processing 9900 batches\n",
      "I1027 10:53:54.308731 139683208804160 csv_writer.py:42] Finished writing\n"
     ]
    }
   ],
   "source": [
    "config = Config(mode=\"predict_on_the_fly\", config_json=config_json)   \n",
    "app = TextClassification(user_defined_config=config)\n",
    "pred_reader = CSVReader(input_glob=app.predict_input_fp,\n",
    "                        is_training=False,\n",
    "                        input_schema=app.input_schema,\n",
    "                        batch_size=app.predict_batch_size)\n",
    "\n",
    "pred_writer = CSVWriter(output_glob=app.predict_output_fp,\n",
    "                        output_schema=app.output_schema)\n",
    "\n",
    "app.run_predict(reader=pred_reader, writer=pred_writer, \n",
    "                checkpoint_path=app.predict_checkpoint_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = pd.read_csv('./data/predict.csv', header=None, delimiter='\\t', encoding='utf8')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred.columns = ['true_label','pred_label_id']"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>true_label</th>\n",
       "      <th>pred_label_id</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>4</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>7</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>0</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>b'agriculture'</td>\n",
       "      <td>10</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "       true_label  pred_label_id\n",
       "0  b'agriculture'              4\n",
       "1  b'agriculture'              4\n",
       "2  b'agriculture'              4\n",
       "3  b'agriculture'              4\n",
       "4  b'agriculture'             10\n",
       "5  b'agriculture'              4\n",
       "6  b'agriculture'              4\n",
       "7  b'agriculture'              0\n",
       "8  b'agriculture'             10\n",
       "9  b'agriculture'             10"
      ]
     },
     "execution_count": 19,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "pred.head(10)"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
